A Heuristic-Regression Approach to Crawler Pattern Identification on Clickstream Data

Web robots, crawlers and spiders are software agents that visit Web sites periodically for multiple purposes. Usually, these activities impel the generation of additional clickstream and pattern data that will rise the necessity for extra processing and filtering. Robots are not conventional Web users. However, some of them intentionally pretend to be so. Their requests flood Web server logs, preventing the discovery of real patterns and trends and forcing the application of additional filtering clickstreams procedures. In order to detect the influence of crawlers confusing usage pattern analysis, we designed a clickstream crawler identification and filtering system that will be described through out this paper. Based on a set of heuristic approaches, we created a filtering mechanism in order to get clean record entries about potential crawlers’ requests. This system generates a confidence factor for each treated record, representing the certainty that it had when identifying the entry as crawler request. Afterwards, a multidimensional database is populated with these records, complemented with the previous confidence factor. This allows the application of regression techniques trying to establish new and more effective crawler behaviour patterns.